Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try to resolve container.name from the injected agent args #1319

Merged
merged 11 commits into from
Dec 1, 2020
Merged

Try to resolve container.name from the injected agent args #1319

merged 11 commits into from
Dec 1, 2020

Conversation

lujiajing1126
Copy link
Contributor

Resolve #1318

@codecov
Copy link

codecov bot commented Nov 27, 2020

Codecov Report

Merging #1319 (5a14178) into master (f250749) will increase coverage by 0.05%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1319      +/-   ##
==========================================
+ Coverage   87.35%   87.41%   +0.05%     
==========================================
  Files          89       89              
  Lines        4999     5021      +22     
==========================================
+ Hits         4367     4389      +22     
  Misses        467      467              
  Partials      165      165              
Impacted Files Coverage Δ
pkg/inject/sidecar.go 94.76% <100.00%> (+0.61%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f250749...5a14178. Read the comment docs.

Copy link
Contributor

@jpkrohling jpkrohling left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pkg/inject/sidecar.go Outdated Show resolved Hide resolved
@@ -378,3 +386,12 @@ func EqualSidecar(dep, oldDep *appsv1.Deployment) bool {
oldDepContainer := oldDep.Spec.Template.Spec.Containers[oldDepIndex]
return reflect.DeepEqual(depContainer, oldDepContainer)
}

func parseAgentTags(args []string) []string {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we follow the refactoring mentioned above, we might not need this. The reason is that we'd have a consistent result for the map, as long as the container name doesn't change. Once it changes, we want this new name to be used in the map/args.

Copy link
Contributor Author

@lujiajing1126 lujiajing1126 Nov 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I partially agree for this point. Suppose we have a deployment with one single container at the very beginning, of course we can readily deduce the container.name. Then we inject several sidecars base on the annotations, let's say, istio-proxy and jaeger-agent, we will finally have three containers in this deployment.

At the second time, even if we are able to filter the jaeger-agent out, we still have two containers from which we are not able to pick the correct container.name without a prior. So I think it is still necessary to parse the Tags / Args from the existing agent container.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should really leave the container.name out if we have more than one container (not counting the Jaeger Agent one). It's certainly possible that traces are coming from both containers in that case, so, the annotation will be wrong. Reusing the container name from an older version of the pod doesn't seem right to me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should really leave the container.name out if we have more than one container (not counting the Jaeger Agent one). It's certainly possible that traces are coming from both containers in that case, so, the annotation will be wrong. Reusing the container name from an older version of the pod doesn't seem right to me.

I understand your argument and I think it is quite reasonable. But the deployment may still probably restart once if we inject more than one sidecar. That would cause a serious problem in a large cluster (we have ~500 deployments in our production cluster). How can we prevent from this situation?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will happen only if you didn't have another container before, but now you have, right? The only way to prevent that, from what I can see, is to add the agent sidecar manually, so that the operator won't manage this container for you.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Specifically, I am concerned with the scenario with two operators: one is jaeger-operator, the other is istio-operator.

Emm, what about adding an annotation to specify the container-name, if the user want to set the name explicitly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Users can do that already by adding env vars to their own container (JAEGER_TAGS). The tags we add to the sidecar container are ones that we add based on information we can infer from the env.

Signed-off-by: Megrez Lu <[email protected]>
@lujiajing1126
Copy link
Contributor Author

lujiajing1126 commented Nov 28, 2020

I try to refactor the code following the comment made by @jpkrohling

  • A default map of tags is created
  • Existing tags are parsed if jaeger-agent is already injected
  • The old map is overwritten by new generated default values
  • The tags are now sorted in alphabetic order to keep it stable

Still a question, do we need to remove container.name if it is not in the latest deployment, or just got changed?

go.mod Outdated
@@ -7,7 +7,6 @@ require (
github.com/ghodss/yaml v1.0.1-0.20190212211648-25d852aebe32
github.com/go-openapi/spec v0.19.8
github.com/googleapis/gnostic v0.3.1
github.com/miekg/dns v1.1.35 // indirect
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's up with the changes to go.mod and go.sum ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reverted all changes to go.mod and go.sum. But may be this is an indirect dependency, so it will be cleaned by go mod tidy?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be. Once this gets merged, would you like to run a go mod tidy against master and seeing if there are differences in the go.mod?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I would like to have a try.

pkg/inject/sidecar.go Outdated Show resolved Hide resolved
pkg/inject/sidecar.go Show resolved Hide resolved
pkg/inject/sidecar_test.go Outdated Show resolved Hide resolved
Copy link
Contributor

@jpkrohling jpkrohling left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. @rubenvp8510, please take a look as well. Once you give your approval, I'll merge this.

assert.Contains(t, agentTags, "container.name=only_container")
agentTagsMap := parseAgentTags(dep.Spec.Template.Spec.Containers[1].Args)
assert.Contains(t, agentTagsMap, "container.name")
assert.Equal(t, agentTagsMap["container.name"], "only_container")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you are changing this PR for some other reason, you might want to remove the Contains, as the Equals assertion already covers it.

Copy link
Contributor Author

@lujiajing1126 lujiajing1126 Dec 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you're right. Unnecessary Contains Removed. Also fix the same issue in two other places.

@lujiajing1126
Copy link
Contributor Author

Another problem may possibly come from the different logic in deployment_controller and namespace_controller, in the latter case we do not check whether the sidecar has already been injected.

But in the deployment_controller, we check that and exclude the container from the existing container list.

IMO, it will also cause unstable status.

@jpkrohling
Copy link
Contributor

Great catch. Would you like to change this PR here to align the logic?

@lujiajing1126
Copy link
Contributor Author

Great catch. Would you like to change this PR here to align the logic?

Sure. And I prefer not to exclude the agent from the container list since we can already reach convergence even if we inject the agent twice. What's your opionion?

@jpkrohling
Copy link
Contributor

I'm not quite sure I understand what you mean, but send the code and I'll review it.

@lujiajing1126
Copy link
Contributor Author

I'm not quite sure I understand what you mean, but send the code and I'll review it.

I have aligned deployment_controller with namespace_controller

@rubenvp8510
Copy link
Collaborator

This looks good to me. Good refactoring!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pod recreation due to container.name tag missing in the jaeger.tags argument
3 participants